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Notes in Brief 

The ever-increasing internationalization of study programs and global 
mobility of students call for greater transparency of and valid information 
on the knowledge and skills students acquire over the course of their 
studies. Several theoretical and methodological challenges arise from the 
immense diversity of degree courses, study programs, and institutions. A 
recent review of the literature has revealed a substantial lack of research 
on assessment practices in higher education, especially on domain-specific 
and generic competency models, as well as on measurement methods 
and valid instruments for competency assessment. The German Federal 
Ministry of Education and Research initiated the national research 
program Modeling and Measuring Competencies in Higher Education 
(KoKoHs) to address these challenges. This article describes the assessment 
practices, aims, and conceptual and methodological framework of 
KoKoHs and presents the main results of the first funding phase of the 

program. 
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-M-Varlier approaches to competency assessment in higher education were limited 
mostly to prerequisite admissions tests, data on learning opportunities, and subjective 
measures (Kuhn & Zlatkin-Troitschanskaia, 2011). Recent analyses have revealed that 
accredited higher-education institutions lack sufficiently reliable and valid instruments to 
assess students’ learning outcomes and that there are tremendous differences in competency 
evaluation in higher education within departments and institutions, as well as among 
institutions nationally and internationally. Hence, not surprisingly, the results of many 
studies indicate that most certificates and grades in higher education are hardly comparable 
on the national level let alone on an international level (Zlatkin-Troitschaskaia, Shavelson 
& Kuhn, 2015). Given the increasing internationalization and global mobility of students, it 
is imperative there is transparency of students’ knowledge and skills across various study 
models and countries. 

Valid assessment of competencies in higher education form the basis for transparency 
and comparability of academic degrees, which are stipulated aims of policy reform programs. 
Therefore, in 2010, the German Federal Ministry of Education and Research (BMBF) initiated 
the national Modeling and Measuring Competencies in Higher Education (KoKoHs) research 
program, which addresses the political and practical challenges of conducting competency 
assessment in higher education. We present an overview of the main outcomes following the 
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end of the first five-year funding phase of the KoKoHs program. First, we outline the structure, 
aims, and theoretical and methodological framework of KoKoIIs. Second, we present the 
accumulated results in the areas of competency modeling, test development, and validation. 
Third, we report on key activities of the program that will shape the future of competency 
assessment in higher education in Germany, including the dissemination of results, 
internationalization of KoKoIIs networks, and provision of support for young researchers. We 
conclude by outlining challenges and perspectives for the second funding phase of KoKoIIs. 

KoKoHs: Structure and Aims 

The KoKoIIs program provides systematic, internationally compatible and fundamental 
research on competency assessment in higher education (Zlatkin-Troitschanskaia, Kuhn, & 
Toepper, 2014). During the first phase, the program included 70 projects with 220 researchers 
at more than 50 institutions of higher education in Germany and Austria. Selected during an 
external review process, each 24 cross-university collaborative project was required to bring 
together domain experts, teaching methodology experts, and research methodology experts 
from at least two universities. KoKoIIs projects involved more than 50 international experts 
(from universities, testing institutes, etc.) from 20 countries including the United States, 
Australia, Japan, and South Korea. The first phase ran from 2011 to 2015. Having received 
positive external evaluation, the program is continued for another five years (2016 to 2020). 

The general purpose of the KoKoIIs program is to model and assess systematically 
domain-specific and generic competencies of students in higher education. KoKoIIs projects 
take into account curricular and job-related requirements, transform theoretical competency 
models into suitable measuring instruments, and validate test score interpretations. To enable 
meaningful cooperation and promote cross-project synergies during the first phase, KoKoHs 
focused on student competencies in one generic cluster (self-regulation and general research 
competencies) and four domain-specific clusters comprising some of the most popular fields 
of study in Germany: 

• engineering, including electrical engineering and mechanical engineering; 

• economics and social sciences, including teacher training in economics and 

social sciences; 

• educational sciences, including psychology; and 

• teacher training in science, technology, engineering, and mathematics 

(STEM subjects). 

Conceptual and Methodological Framework 

In the KoKoIIs projects, competencies were defined as latent cognitive and 
noncognitive underpinnings of performance (Ewell, 2005; Rychen, 2004). The KoKoIIs 
program adopted Weinert’s (2001) definition of competencies as “cognitive abilities and skills 
that individuals possess or acquire in order to solve certain problems as well as the aligned 
motivational, volitional and social dispositions and skills to apply the solutions in different 
situations successfully and responsibly” (p. 27-28). This general definition was specified for 
competencies acquired in higher education. During the first phase, KoKoHs projects focused 
predominantly on (latent) cognitive abilities and skills and specified them for their respective 
fields of study (Alexander, 1997; Alexander, Winters, Dinsmore, & Parkinson, 2011). 

Models of knowledge and skills were operationalized through measuring instruments 
and tested in empirical assessments. Validation efforts aimed to establish validity of 
interpretation and answer the key question: What can we infer from the (cognitive) 
representations elicited by the assessment of the actual competencies of students? This 
approach is always challenging: The underlying cognitive abilities and skills—ideally 
also the corresponding noncognitive (e.g., affective-motivational) aspects—need to be 
operationalized through representative, practice-oriented, and often domain-specific tasks; 
assessments need to represent specific situational contexts and be free of potential bias, 
such as measurement errors or influences of construct-irrelevant test-taking behavior 
(Kulikowich & Alexander, 1994; 2003). 


Given the increasing 
internationalization 
and global mobility of 
students, it is imperative 
there is transparency of 
students’ knowledge and 
skills across various study 
models and countries. 
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KoKoHs projects take into 
account curricular and 
job-related requirements, 
transform theoretical 
competency models 
into suitable measuring 
instruments, and validate 
test score interpretations. 


The general assessment framework in KoKoHs was based on the Assessment Triangle 
by Pellegrino, Chudowsky, and Glaser (2001), which covers three fundamental aspects of 
assessment: “a model of student cognition and learning in the domain, a set of beliefs about 
the kinds of observations that will provide evidence of students’ competencies, and an 
interpretation process for making sense of the evidence”(p. 44) (see also Shavelson, 2013; 
Webb, Shavelson, & Steedle, 2012). These three aspects corresponded with key objectives 
of KoKoHs: 

1. Define the construct to be assessed ( cognition ). 

2. Develop and use suitable models and measuring instruments ( observation ). 

3. Draw valid inferences from the assessment data ( interpretation ). 


The Assessment Triangle provided the cornerstones for an assessment connecting 
theoretical constructs of students’ competencies with empirical evidence; that is, developing 
estimates based on limited instances of students’ knowledge and skills in an argument- 
based approach of “reasoning from evidence” (Mislevy, 1994). For more specific, practical 
orientation, KoRolls project teams adopted the evidence-centered assessment approach and 
test development concept (Mislevy & Ilaertel, 2006; Hattie, Jaeger, & Bond, 1999), which 
includes the following steps: 


• Domain analysis and modeling: In the assessment of competencies in 
higher education, initial steps included analyzing and defining the domain 
and modeling the domain-specific construct to be assessed. 

• Assessment framework: An assessment framework was defined, which 
served to operationalize the theoretical model and develop items for the 
test instruments. 

• Assessment implementation: The instruments were tested empirically. 

• Assessment delivery: The test scores were analyzed using various 
psychometric models. Analyses always included evaluations of fit of the data 
to the theoretical constructs and to the corresponding data interpretations. 

The conclusive evaluation of the tests with regard to various validation 
criteria served as a basis for further decisions (see also Pant, Rupp, 

Tiffin-Richards, & Roller, 2009). 

Validation is of paramount importance in KoKoHs. Validation efforts followed the International 
Standards for Educational and Psychological Testing (AERA, APA, & NGME, 2014; Kane, 2013). 

Overview of the Main Results 

The following is a synopsis of the results of the KoKoHs program at the end of the first 
funding phase and before the beginning of the second funding phase. Results are summarized 
and presented for the three main areas of work in the program: development of competency 
models; development of test instruments; and validation. Furthermore, we describe the efforts 
taken to reach the following three strategic aims of the program: to achieve national and 
global visibility; to ensure internationalization and compatibility of results; and to help young 
researchers establish a specialized research community. 

Competency Models, Assessments, and Validation 

The teams of the 24 collaborative projects developed 41 competency models of generic 
and domain-specific competencies in higher education. Content validity (including curricular 
validity) was ensured in the KoKoHs projects through analyses of almost 1,000 documents 
such as module descriptions and study regulations from more than 250 institutions of higher 
education throughout Germany. Furthermore, analyses of items and tasks from almost 1,500 
documents (e.g., exams, exercises, lecture notes) informed the construction of test items as 
shown in Table 1. 
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Table 1 


Competency Models Developed and Validation 


Theoretical competency models 

41 

Document analyses of 

curricula, regulations, standards 

910 

exams, exercises, lecture notes 

1,350 

project and lab reports 

48 

Validation 

expert interviews 

556 

cognitive labs 

459 


In addition to these document analyses, validation measures employed in the KoKoIIs 
projects often included expert interviews and cognitive labs (with N-500 experts and N-500 
participants, respectively, across the 24 collaborative projects). Expert interviews provided 
evidence of content validity; cognitive labs provided evidence of cognitive validity through 
analyses of fit between cognitive processes in the theoretical models and thought processes 
observed empirically in think-aloud interviews while participants responded to items. 


Table 2 

Instruments Developed and Competency Assessments Conducted 


Instruments 

paper-pencil tests 63 

computer-based tests 36 

video-based tests 8 

other tests (e.g., critical incidents) 119 

Assessment surveys 

institutions 226 

participants 49,904 


The teams of the 24 collaborative projects also created new assessment instruments 
based on the competency models developed, and/or they adapted existing international 
instruments, if available, to meet their needs. Altogether, more than 60 paper-pencil instruments 
and almost 40 computer-based instruments were developed in the KoKoIIs program as 
depicted in Table 2. During the first five-year funding phase, more than 220 researchers and 
several hundred student assistants were involved in project work. In addition, lecturers and 
students from the participating institutions in Germany actively supported the program by 
supervising or participating in the surveys and assessments during classes. As an incentive, all 
participating institutions, lecturers, and students received professionally prepared feedback 
from the aggregated, anonymized data. In turn, this facilitated transfer of research results and 
findings into higher-education practices. 


Additional, more action-oriented approaches were employed in the KoKoHs projects 
for valid assessment of specific competency facets. For example, video-recorded role plays 
were used to assess explanatory knowledge of pre-service physics teachers within the 
domain of teacher education. The KoKoIIs reseachers used—in addition to almost 10 newly 
developed video-based instruments—various other measuring methods, such as critical 
incidents, for complementary, qualitative, in-depth analyses of competency levels. Moreover, 
the teams of the KoKoHs collaborative projects used quantitative methods, (e.g., structural 
equation analysis) to gather evidence of validity aspects such as the internal structure of 
competency constructs. 
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Overall, the teams of the 
KoKoHs collaborative 
projects assessed domain- 
specific and generic 
competencies as well as 
personal and structural 
influence factors of 
approximately 50,000 
students at more than 
220 institutions of higher 
education in Germany as 
well as in Austria 


To enhance global 
visibility and 
international 
networking as well as 
to ensure compatibility 
with international 
research and higher 
education practices, 
KoKoHs researchers 
established and 
maintained cooperation 
with international 
experts in different 
research areas. 
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The internal structure was differentiated according to both content requirements (e.g., 
knowledge and skills related to financial plans as part of business competency) and cognitive 
requirements (e.g., remembering, applying, or evaluating) (Anderson & Krathwohl, 2001). 

Overall, the teams of the KoKoHs collaborative projects assessed domain-specific and 
generic competencies as well as personal and structural influence factors of approximately 
50,000 students at more than 220 institutions of higher education in Germany as well as in 
Austria. Results in the three areas of competency modeling, test development, and validation 
significantly contributed to the provision of a reliable, valid, and internationally compatible 
basis for competency assessment in higher education in Germany. 

Project Example 

The WiwiKom project, which focuses on modeling and measuring the competencies 
of students and graduates of business and economics, provides one example of how the 
conceptual and methodological framework was implemented and how psychometric validity 
was gathered (Zlatkin-Troitschanskaia, Forster, Bruckner, & Happ, 2014). The construct 
of professional competency in business and economics was defined in a theory-driven 
competency model based on Kane’s interpretative use argument (2013). Empirical evidence 
gathered in the assessments was described in the validity argument; subsequently, analyses of 
the data indicate the modeling was adequate (Kane, 2013). 

The theory-driven model of competency in business and economics developed 
in WiwiKom (Zlatkin-Troitschanskaia et al., 2014) differentiated seven domain-specific 
content dimensions and three levels of cognitive requirements. The content dimensions 
represented the core curriculum in business and economics, sub-divided into content areas 
(e.g., microeconomics, finance, etc.). The cognitive dimension specified levels of professional 
competency defined in terms of the mental processes (e.g., understanding, applying, etc.) 
necessary to respond appropriately to situational cognitive requirements of increasing 
complexity. The competency model served as a basis for developing the WiwiKom test 
instrument and validating it in qualitative and quantitative studies with a focus on the five key 
validity aspects, while adhering to the Standards for Educational and Psychological Testing 
(AERA et al., 2014). 

For curricular and content validation, the test content was examined during document 
analyses and was compared to curricula and textbooks from 98 degree courses at 64 institutions 
of higher education in Germany; it also was evaluated by lecturers of business and economics 
during expert interviews (N=32) and in online ratings (N=78). For cognitive validation, mental 
processes of 32 students were examined in cognitive labs, where students were asked to 
think aloud while responding to test items. For item calibration, test standardization, and 
establishment of validity of internal test structure, three field surveys were conducted in the 
WiwiKom project, assessing approximately 10,000 students of business and economics from 
all years of study at 57 institutions of higher education in Germany. The data was analyzed 
using methods such as confirmatory factor analysis or IRT modeling to gather evidence on 
the dimensionality and gradation of the examined competency. In addition, surveys were 
administered to gather data on multiple personal variables (e.g., gender, prior knowledge, etc.) 
and institutional variables (e.g., type of institution) for analyses of the relationship between 
the construct and other variables. 

Further Activities of the KoKoHs Program 

In addition to the specific research goals, there were three strategic aims of the 
KoKoHs program which would define the long-term impact of the program. 

National and Global Visibility 

A major aim of KoKoHs project was to achieve national and global visibility through 
the dissemination of our results. The teams of the KoKoHs collaborative projects were highly 
productive, primarily publishing articles in high-ranking national and international journals. 
Moreover, approximately 250 presentations were held at national conferences, and almost 100 
presentations were held at high-profile international conferences, such as annual meetings 
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of the European Association for Research on Learning and Instruction (EARLI) and the 
American Educational Research Association (AERA). 

In addition to the project teams presenting and publishing the results of individual 
projects, results related to the entire KoRolls program also were documented and published 
by the coordination project. The coordination project not only contributed numerous 
presentations and posters to national and international conferences with a focus on scientific 
topics or higher-education practice as shown in Table 3, but also published its own KoKoHs 
Working Papers series (with seven issues altogether, five of which were in English 1 ) as well 
as seven thematic issues in prestigious national and international journals, some of which 
were coedited by renowned international cooperation partners. The KoKoHs program is the 
only national research program in Germany that has published an overview paper and is 
represented in an international edited volume on all research initiatives worldwide in the area 
of learning outcomes assessment in higher education. 

Table 3 

Project Results Disseminated 
Publications 


national 

134 


international 

65 


Presentations 



national 

244 

The KoKoHs program has 

international 

89 

more than 50 international 



cooperation partners 



from 20 countries on four 



continents. 


Internationaliz ation 


To enhance global visibility and international networking as well as to ensure 
compatibility with international research and higher education practices, KoKoHs researchers 
established and maintained cooperation with international experts in different research areas. 

International KoKoIIs cooperation partners include experts from various universities, research 
associations, and public and non-profit higher education and research institutions, including 
testing institutes. The KoKoIIs program has more than 50 international cooperation partners 
from 20 countries on four continents. During the first funding phase, cooperation between 
KoKoHs project teams and international partners included joint events such as the KoKoIIs 
Affiliated Group Meeting at the 2014 AERA conference in Philadelphia (Kuhn, Toepper, & 

Zlatkin-Troitschanskaia, 2014), joint publications such as a special issue in the journal Studies 
in Higher Education (Zlatkin-Troitschanskaia & Shavelson, 2015), and in the Peabody Journal 
of Education (Zlatkin-Troitschanskaia, Blomeke, & Pant, 2015) as well as joint supervision of 
doctoral and post-doctoral projects of KoKoIIs researchers. 

Supporting Young Researchers 

With approximately 70 doctoral projects and almost 20 post-doctoral projects 
conducted by KoKoIIs researchers, a major focus of the program was to systematically support 
young researchers in building up a scientific community within empirical higher-education 
research in Germany. Providing the young researchers with the necessary guidance 
would enable them to close existing gaps in research on competency assessment in higher 
education. To this end, the KoKoHs coordination project organized for all young researchers 
a variety of systematic training opportunities and events throughout the course of the 
program including methodology workshops, mentoring, and networking events such as 
international colloquia. Workshops were held at regular intervals over the course of the 
program on various 

1 KoKoIIs Working Papers can be downloaded at http://www.kompetenzen-im-hochschulsektor.de/ 617_ 

DEU_IITML.php. See the KoKoIIs homepage in English for more in-depth information, including details 
about KoKoIIs events. 
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topics related to research methodology, including a general introduction to methods of social 
research, item and test development, scaling and test theory, validation, and longitudinal data 
analysis. Networking and mentoring events such as the International Colloquium for Young 
Researchers in November 2013 and the international Autumn Academy in October 2014 
were organized for outstanding young researchers whose submissions had been selected by 
international experts (Toepper, Zlatkin-Troitschanskaia, Kuhn, Schmidt, & Bruckner, 2014). 

These events presented young researchers with excellent opportunities for networking 
internationally, presenting their work to the international scientific community, and receiving 
feedback from renowned international experts. Further opportunities for international 
networking and exchange open to all researchers in the program included the international 
kick-off and closing conferences as part of the cooperation between individual collaborative 
projects and international partners. 


KoKoHs program 
goals for the second 
funding phase include 
maintaining and 
expanding the networks 
established thus far, while 
continuing to support and 
draw on the expertise of 
the research community 
solidified in Germany 
during the first phase. 


Conclusions and Future Perspectives 

During the first funding phase, KoKoIIs addressed theoretical, methodological, 
and empirical challenges including: systematically designing or adapting tests; considering 
framework conditions such as time, method, and format; analyzing data with complex 
psychometric methods; confirming psychometric quality criteria; and undertaking 
comprehensive validation. The models of competency structures and levels, the assessment 
designs, and the measuring instruments developed and tested so far provide a solid basis for 
future in-depth longitudinal multilevel analyses in random field experimental studies in higher 
education. 

To date, few studies in higher education have employed complex methodological 
designs, such as longitudinal modeling, multilevel modeling, or (quasi-)experimental designs. 
Hence, findings on the trajectory of competencies over the course of studies are still scarce. 
With regard to instruments, there remains a lack of innovative measurement methods such as 
adaptive computer-based testing. Many challenges need to be addressed in order to overcome 
the unsatisfactory state of having to rely on less direct indicators (i.e., grades, degrees, and 
students’ self-evaluations) and to complement these existing indicators with more direct 
assessments that allow valid conclusions to be drawn about student competencies (Zlatkin- 
Troitschaskaia, Shavelson, & Kuhn, 2015). 


In 2015, the BMBF launched the second phase of the KoKoIIs research program. The 
remaining theoretical, methodological, and empirical challenges will be addressed in this 
funding phase. These challenges include systematically designing or adapting tests under 
time, method, and format constraints, analyzing data with complex methods, confirming 
psychometric quality criteria, and undertaking comprehensive validation (AERA et al., 2014). 
Due to specific challenges in higher education—reliability issues related to complex models 
constrained by limited testing time, panel mortality in longitudinal studies, and testing based 
on students’ performance—more complex and innovative methods of analysis need to be 
considered. These methods include longitudinal and multilevel analyses in random field- 
experimental studies, adaptive computer-based testing, and suitable psychometric techniques. 
KoKoIIs program goals for the second funding phase include maintaining and expanding 
the networks established thus far, while continuing to support and draw on the expertise 
of the research community solidified in Germany during the first phase. More systematic 
international collaboration and exchange of best practices from this field (and related areas 
such as competency assessment in the school sector) are needed. 
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